System and Method for Automatically Analyzing Phenotypical Responses of Cells

ABSTRACT

A system and a method to analyze a phenotypical response of cells to a treatment are disclosed in which a model development module receives images of a plurality of reference cell carriers and treatment information associated with the plurality of reference cell carriers, identifies parameters of cells in the image that distinguish those reference cell carriers to which the treatment has been applied from other reference cell carriers, and trains a model using the identified parameters. A high-content imaging system includes an image capture device, and the image acquisition module receives from the image capture device a plurality of images of cell carriers to be evaluated. The model application module applies the trained model to the plurality of images of the cell carriers to be evaluated to predict a concentration of the treatment applied to each of the cell carriers evaluated.

RELATED APPLICATIONS

The present application is a continuation of Cohen et al., U.S. patentapplication Ser. No. 15/750,579 filed on Feb. 6, 2018; which is a U.S.National Phase Entry of Cohen et al., International Patent ApplicationNo. PCT/US2016/045784, filed Aug. 5, 2016; and which in turn claimsbenefit of Cohen et al., U.S. Provisional Patent Appl. No. 62/204,225,filed Aug. 12, 2015. All of these applications are entitled “System andMethod of Automatically Analyzing Phenotypical Responses of Cells” andthe entire contents of these applications are incorporated herein byreference.

FIELD OF DISCLOSURE

The present subject matter relates to high content imaging systems, andmore particularly, to a system and method to automatically analyzephenotypical responses of cells.

BACKGROUND

A researcher may use a high content imaging system (HCIS) to analyzephenotypical responses of biological cells to a treatment, such as, forexample, exposure to particular substances and/or other environmentalchanges. Such responses may be visually distinct and may be observablein images acquired using fluorescent labeling and other techniques.

However, such responses may vary in accordance with the pathway involvedin the response, cell tissue type, cell age, growing condition, and thelike. The variety of possible responses requires the analysis of theimages of cells be limited to a particular type of phenotypical responsesuch as, for example, translocation of a protein from one location in acell to another, expression or presence of a particular molecule,congregation or clumping of a molecule, and the like. Further, each typeof phenotypical response that may be analyzed may be associated with aplurality of adjustable parameters that may be measured in images ofsuch cells, and the presence or absence of such response may beindicated by a magnitude of such measured parameters. The quantity ofthe types of responses and the number of parameters that must beadjusted to quantify each type of response may pose a significant burdenon a researcher because the researcher must identify the phenotypic typeof response that occurs, and then manipulate the various parametersassociated with such type to determine if such response did indeedoccur.

Further, in some cases, it may not be practical or even possible toexpose a homogenous population of target cells to a substance orenvironmental condition to be tested. In such cases, a heterogeneouspopulation of cells that includes the target cells are exposed to thesubstance or environmental condition, images of the heterogeneouspopulation are acquired, and such images have to be analyzed todetermine if the particular target cell in the heterogeneous populationexhibits a particular phenotypical response.

SUMMARY

According to one aspect, a system to analyze a phenotypical response ofcells to a treatment includes an image capture device and a responseanalysis system. The response analysis system includescomputer-executable code stored on one or more non-transitory storagedevices that, when executed, causes one or more processors to receiveimages of a plurality of reference cell carriers and treatmentinformation associated with the plurality of reference cell carriers andidentify parameters of images of the plurality of reference cellcarriers based on the treatment information, the identified parametersdistinguishing those cells in the plurality of reference cell carriersto which the treatment has been applied from other cells in theplurality of reference cell carriers. In addition, thecomputer-executable code causes the one or more processors to train amodel using the identified parameters, receive from the image capturedevice an image of a cell carrier to be evaluated, and apply the trainedmodel to the image of the cell carrier to be evaluated to indicate aresponse level of a cell in the cell carrier to be evaluated.

According to another aspect, a method to analyze a phenotypical responseof a cell to a treatment includes the steps of receiving images of aplurality of reference cell carriers and treatment informationassociated with the plurality of reference cell carriers and identifyingparameters of the images and/or cells represented in the images based onthe treatment information, the identified parameters distinguishingthose cells in the plurality of reference cell carriers to which thetreatment has been applied from other cells in the plurality ofreference cell carriers. The method also includes the steps of traininga model using the identified parameters, receiving from an image capturedevice an image of a cell carrier to be evaluated, and applying thetrained model to the image of the cell carrier to be evaluated toindicate a response level of a cell in the cell carrier to be evaluated.

Other aspects and advantages will become apparent upon consideration ofthe following detailed description and the attached drawings whereinlike numerals designate like structures throughout the specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a high content imaging system;

FIG. 2 is a block diagram of a response analysis system that may be usedwith the high content imaging system of FIG. 1;

FIGS. 3 and 4 illustrate sample trays that may be used with the responseanalysis system of FIG. 2;

FIG. 5 is a flowchart of steps undertaken by the response analysissystem of FIG. 2 to develop a model; and

FIG. 6 is a flowchart of steps undertaken by the response analysissystem of FIG. 2 to apply the model developed in accordance with theflowchart of FIG. 5.

DETAILED DESCRIPTION

Referring to FIG. 1, as will be apparent to those who have skill in theart, an HCIS 100 may include an X-Y stage 102, one or more objectivelenses 104, one or more illumination sources 106, one or more filters108, an image capture device 110, and a controller 112. The HCIS 100 mayalso include one or more mirrors 114 that direct light from theillumination source 106 to a sample tray 116 that may be disposed on theX-Y stage 102, and from such sample tray 116 to the image capture device110. Typically, the sample tray 116 includes a plurality of wells 118,and samples (for example, biological cells) to be imaged by the HCIS 100may be disposed in each such well 118.

Although, FIG. 1 shows the light from the illumination source 106reflected from sample tray 116 reaching the image capture device 110, itshould be apparent that additional mirrors (not shown) may be used sothat light from the illumination source 106 is transmitted through thesample tray 116 and directed toward the image capture device 110.Further, it should be apparent that in some cases no illumination fromthe illumination source 106 may be necessary to image the samples in thesample tray 116 (for example, if the samples emit light or if thesamples include radioactive components). In some embodiments, light fromthe illumination source may be transmitted through the samples in thesample tray 116, and the samples refract and/or absorb the transmittedlight to product lit that is imaged.

During operation, the sample tray 116 may be placed, either manually orrobotically, on the X-Y stage 102. In addition, the controller 112 mayconfigure the HCIS 100 to use a combination of a particular objectivelens 104, illumination generated by the illumination source 106, and/orfilter 108. For example, the controller 112 may operate positioningdevices (not shown) to place a selected objective lens 104 and,optionally, a selected filter 108 in the light path between the sampletray 116 and the image capture device 110. The controller 112 may alsodirect the illumination source 106 to illuminate the sample tray 116with particular wavelengths of light. The samples in the sample tray 116may contain molecules that fluoresce, either naturally occurringmolecules, or molecules produced or present within the samples due totreatment. The wavelength illuminating the sample may be the excitationwavelengths associated with such fluorescent molecules, and the imagingcapture device will capture only the emission spectrum of suchfluorescent materials. One or more wavelengths may used serially orsimultaneously to illuminate the same samples and produce images

In addition, in some embodiments, the controller 112 may operate a focusmechanism 120 so that the image capture device 110 may obtain in-focusimages of samples disposed in the sample tray 116.

Thereafter, the controller 112 may operate the X-Y stage 102 so that awell 118 or a portion thereof is in a field of view of the image capturedevice 110, and actuate the image capture device 110 to capture an imageof the well 118 or the portion thereof. The controller 112 mayrepeatedly operate the X-Y stage 102 and the image capture device 110 inthis manner until images have been captured of all of the wells 118 ofthe sample tray 116 that are of interest. Further, the controller 112may capture several images of the same well 118 or portion thereof,wherein each such image is captured using a different combination of oneof the objective lenses 104, one or more of the filters 108, andillumination generated by the illumination source 106.

Referring to FIGS. 1-4, a response analysis system 200 may be used withthe HCIS 100 to automatically determine if any cells disposed in thewells 118 of the sample tray 116 are responsive to a treatment. Areference sample tray 116A is prepared with identical populations ofsample cells disposed in each well 118A thereof. The population disposedin each well 118A may be a homogeneous population that comprises onlytarget cells, or a heterogeneous population that is a combination oftarget cells and non-target cells. Thereafter, selected wells 118A ofthe reference sample tray 116A may be subjected to a treatment.

In some embodiments, selected wells 118A of the reference sample tray116A may either have a predetermined dose or concentration of thetreatment applied thereto, and other wells 118A may have no treatmentapplied thereto. As illustrated in FIG. 3, the wells 118A in columns 1-6may have been subjected to the treatment, as indicated by the “+”character, and the wells 118A in columns 7-12 may not have beensubjected to the treatment, as indicated by the “−” character.

In some embodiments, the wells 118A of the reference sample tray 116Amay have different dosages or concentrations of the treatment appliedthereto. As illustrated in FIG. 4, the wells 118A in rows A-C andcolumns 1-3 have not had any treatment applied thereto, as indicated bythe “−” character. The wells 118A in rows A-C and columns 4-6 have had afirst concentration, for example, one unit of the treatment appliedthereto as indicated by the “1” character, the wells 118B in rows A-Cand columns 7-9 have had a second concentration, for example, two unitsof the treatment applied thereto as indicated by the “2” character. Inthe present example shown in FIGS. 3 and 4, each unique character shownin the wells 118A of the reference sample tray 116A is associated with adifferent concentration of the treatment applied to such wells.

An operator uses the user computer 202 to specify treatment informationto the response analysis system 200, and such treatment informationincludes an indication of the concentration of the treatment applied toeach well 118A of the reference sample tray 116A. A user interfacemodule 204 receives such reference sample tray information and storesthe information in a treatment information database 206. In someembodiments the user interface module 204 allows the operator to uploada file, for example, a text file, a spreadsheet file, or other data filethat specifies, for each well 118A of the reference sample tray 116A, aconcentration or amount of the treatment applied to such well 118A. Inother embodiments, the user interface module 204 displays on the usercomputer 202 a representation of the wells 118A of the reference sampletray 116A, and allows the operator to use an input device (not shown) ofthe user computer 202 to select a representation of each well, andindicate a concentration of the treatment applied to each well 118Aassociated with such representation. Other ways apparent to those whohave skill in the art may be used to supply the reference sample trayinformation to the response analysis system 200.

Thereafter, an image acquisition module 208 directs the HCIS 100 tocapture a plurality of images of each well 118A, wherein each image iscaptured with a different combination of the objective lens 104,illumination generated by the illumination source 106, and the filter108, as described above. In some embodiments, such plurality of imagesmay be acquired using different modalities, including transmitted light,fluorescence, differential interference contrast, phase contrast,brightfield imaging, and the like.

In some embodiments, the image acquisition module 208 transmits eachparticular combination to the controller 112 of the HCIS 100, and inresponse receives a captured image from the image capture device 110.The received image is then stored in an images database 210. In otherembodiments, the image acquisition module 208 may be integrated with thecontroller 112 of the HCIS 100, and automatically configures the HCIS100 with each combination, directs the image capture device 110 tocapture an image, and stores the image in the images database 210. Insome embodiments, the combinations of objective lens, illumination, andfilter used to capture images are predefined. In other embodiments, theuser may select such combinations in accordance with characteristics ofthe samples in the sample tray 116A. Such characteristics may includemolecules in such samples, the size of the structures of interest in thesamples, and the like.

After all of the images have been captured and stored in the imagesdatabase 210, a model development module 212 analyzes the stored imagesof the reference sample tray 116A to develop a model. Thereafter thedeveloped model may be used to automatically analyze images of othersample trays 116 to determine if the cells in such trays exhibit aphenotypical response to the treatment.

To develop the model, the model development module 212 may use thetreatment information and randomly select a training subset of the wells118A of the reference sample tray 116A. The training subset may includeone or more wells 118A to which no treatment has been applied, and atleast one well 118A for each of the different concentrations of thetreatment that is applied to the sample tray 116A. For example, if thereference sample tray 116A illustrated in FIG. 4 is used, the trainingsubset may include at least one well 118A to which no treatment has beenapplied, and at least one well 118A associated with each of thedifferent concentrations 1-7 of the treatment applied to the wells 118Aof the reference sample tray 116A.

The model development module 212 may provide one or more images of eachwell 118A to a parameter calculation module 214 for analysis. Theparameter calculation module 214 analyzes the images and calculatesvalues of various characteristics or parameters of such images. Forexample, the image may be analyzed to determine an average pixelintensity value thereof, values of pixel coordinates of such image wherecell nuclei are located, a value that represents the strength of pixelintensity variation at edges of cells in the image, a value thatrepresents an orientation of edges relative to nuclei of cells in theimage, a value that represents a measure of texture or variation ofpixel intensities at different magnifications, and the like. Further,the measure of texture may include calculating a standard deviation ofpixel intensity, identifying frequencies associated with peaks of aFourier transform of the image, wavelet matches, and Laplacian ofGaussian peak analysis.

In some embodiments, the parameter calculation module 214 may segmentthe images to identify cell structures associated with each cell nucleiin such images. For example, cell boundaries associated with a cellnuclei may be identified, using one or more images taken with differentwavelengths of light. The area of each cell bounded by such boundariesmay be calculated. Also, the model development module 212 may evaluate ashape of such cells (e.g., circular, elongated, polygonal, etc.).Further, the identified boundary of a cell may be used to mask the imageto isolate structures that are associated with the cell, and values ofparameters described above also may be calculated on such masked image.

In some embodiments, the parameter calculation module 214 may direct theuser interface module 204 to display an image or a portion thereof onthe user computer 202, and to obtain from the operator identification ofpixels of the image (or portion thereof) that are associated with a celland/or cell nuclei. Such operator provided information may be used toautomatically identify boundaries associated with such cell nucleiand/or identify additional cell nuclei. It should be apparent that theparameter calculation module 214 may use filtering, machine learning,and image analysis techniques known to those of ordinary skill in theart to calculate the values the various parameters associated with theimages of the wells 118A of the reference sample tray 116A.

In some embodiments, the parameter calculation module 214 may calculatevalues of parameters of all of the samples or cells in each well 118A orthe reference sample tray 116A. Because the number of wells 118Acompared to the number of samples is relatively low and the number ofmeasurable parameters is relatively large, calculating parameters forthe samples within the whole well 118A may facilitate developinginferences of how a treatment applied to the well 118A is manifested inthe parameters calculated for such well 118A.

In some embodiments, the parameter calculation module 214 calculatesvalues of the parameters that represent a measure of individual cellsidentified in the image of the well 118A such as, for example, adistance from a cell edge to a cell nucleus, cell shape, intensityvariation within the cell, and the like. Further, the parametercalculation module 214 may use information from several images of asample area of a particular well 118A, for example, taken usingdifferent illumination to facilitate calculations of the values of theparameters associated with a particular image of the well 118A.

Cells within the image could be identified various ways. In oneembodiment, the parameter calculation module 214 identifies stainednuclei in each image. Such identification may use a machine-learningalgorithm known to those of ordinary skill in the art such as used byMiniMax, or an intensity and size based algorithm. MiniMax may also beused to identify label free cells using transmitted light. To determinecharacteristics of parameters associated with the well 118A, exactidentification of each cell in such well may not be required. However,if individual cells are identified by the parameter calculation module214, the parameter calculation module 214 may be able to developcharacteristics of parameters associated with individual cells in thewell 118A in addition to characteristics parameters associated with thewell 118A.

In some embodiments, sections of the image of the well 118A may be splitinto sections and “cell like” areas in each such sections could beidentified either by features such as texture or by machine learningalgorithms, as described above.

After values of the parameters have been developed from the images ofthe wells 118A in the subset selected by the model development module212, the model development module 212 statistically correlates thevalues of the parameters associated for each well 118A with thetreatment concentration actually applied to such well 118A. As notedabove, information regarding the treatment concentration actuallyapplied to the well 118A is available from the treatment informationdatabase 206.

For example, if the reference sample tray 116A includes 100 wells thathave a positive treatment and 100 wells that have a negative treatment,and the 80 of the positively treated wells are associated with imagesthat have a relatively high average image intensity and only 20 of thenegatively treated wells are associated with images that have arelatively high intensity, then average image intensity may beconsidered a parameter that is correlated with treatment, and a highvalue of such parameter may be considered correlated with positivetreatment.

In some embodiments, a statistical correlation coefficient, for example,a Pearsons correlation or a Spearman correlation, may be calculatedbetween the value of each parameter associated with the well 118A and/orcell disposed in such well 118A, and the treatment concentration appliedto the well 118A. In other embodiments, the number of correctpredictions of the treatment concentrations associated with eachparameter may be determined. Other ways to determine a correlationbetween the parameter of images associated with the well 118A and thetreatment applied to such well 118A apparent to those of ordinary skillin the art may be used. Further, such correlations may be developed forone or more cells identified in such image.

After the parameters have been correlated, the model development module212 selects those parameters that have the highest correlation with orbest predict the treatment concentration applied to the training subsetof wells 118A.

The model development module 212 may use the values of the selectedparameters and machine-learning techniques to train a model that may beused to identify and/or quantify a response to treatment by cells inwells 118 of a production sample tray 116.

In one embodiment, the model development module 212 may train anartificial neural network in which each input node is associated withone of the selected parameters. The value of each parameter calculatedfrom images of a well 118A of the training subset is provided at theinput node associated with such parameter. The values at the input nodeare propagated through the artificial neural network to produce a valueat the output node of the artificial neural network that predicts theconcentration of treatment applied to such well 118A. The error betweenthe predicted concentration of the treatment and the concentration ofthe treatment actually applied is calculated, and the weights of theinterior nodes of the neural network are adjusted in accordance withsuch error, for example, using backward propagation of the error. Theartificial neural network may be iterated using values at the inputnodes of the parameters calculated from images of additional wells 118Aof the training subset until the weights of the artificial neuralnetwork converge, and the error between predicted treatmentconcentrations of the wells 118A of the training subset and actualtreatment concentrations applied to such wells 118A is minimized.

In other embodiments, a multiple layered neural network may be developedand deep learning methods may be used to train such network. Forexample, values associated with values of complex sets of parameters ofthe images of wells (or cells) may be provided to an input node of sucha network to improve the initial variable weighting of the network. Insome embodiments, a forest tree or random tree forest technique may beused in which the values of the parameters associated with each well118A or cell disposed in the well 118A are analyzed to determine a setof parameters that generates a positive indication of treatment forthose wells 118A of the training subset, or cells disposed in such wells118A, that have had treatment applied thereto.

Metaheuristic techniques may be used to develop the model in which, forexample, a genetic algorithm in which genes are associated withparameters, and the values for such parameters are coded in strands ofsuch genes. Such strands of genes may be used to predict the treatmentapplied to the wells 118A of the training subset, and/or cells disposedin such wells 118A. The fraction of strands that best separate thepositively treated wells 118A and/or cells from the negatively treatedwells 118A and/or cells may be “mated” by interchanging genes betweenpairs of such strands. Over multiple iterations of such mating, lowerscoring strands may be removed so that only the fittest strands survive.The trained model may comprise one such fittest (or highest scoring)strand that survives.

In still other embodiments, the model may be implemented using othermachine learning techniques apparent to those who have skill in the artincluding decisions trees, support vector machines, k-means clustering,swarm, ant colony optimization, and simulated annealing. In someembodiments, as will be apparent to those of ordinary skill in the art,gradient descent and similar techniques may be used to speed convergenceof the machine learning techniques described herein.

The parameters that comprise the inputs of the model developed by themodel development module 212 are stored in a model database 216.

Thereafter, a model evaluation module 218 selects wells 118A thatcomprise an evaluation subset from those wells 118A of the referencesample plate 116A not in the training subset. Like the training subset,the wells 118A that make up the evaluation subset have at least one well118A associated with each concentration of the treatment applied to thereference sample tray 116A. For each well 118A in the evaluation subset,the model evaluation module 218 provides the images from the imagesdatabase 210 to the parameter calculation module 214. The parametercalculation module 214 analyzes such images to calculate values of theparameters identified by the model development module 212 as beingcorrelated with concentration of treatment. The model evaluation module218 then uses the calculated values as an input to the model stored inthe model database 216, and calculates an error between the output ofsuch model with the actual treatment applied to the well 118A. The modelevaluation module 218 calculates such error value for each well 118Athat comprises the evaluation subset. The error values are combined todevelop a confidence score for the model stored in the model database216.

In one embodiment, the error values for each well 118A may bestatistically analyzed to evaluate how well the model predicts if anytreatment has been applied to each well 118A or an amount of thetreatment applied to each well 118A of the evaluation subset. Ifdifferent amounts of treatment are applied to each well 118A of theevaluation subset, the evaluation module 218 may analyze the error barsassociated with a relationship (e.g., a curve) between the amount oftreatment actually applied and the amount of treatment predicted by themodel.

In some embodiments, the evaluation module 218 may compare the amount ofresponse exhibited by cells from previous experiments to the amount ofresponse to treatment identified by the developed model. For example,suppose the treatment includes adding a toxic substance to the wells118, and a dilution series may run among the wells 118. All of the cellsin wells 118 exposed to a high concentration of the toxic substance maybe killed, whereas cells in wells 118 exposed to a low concentration ofthe toxic substance may not show any perceptible change. Experimentaldata that includes images of cells exposed to different concentrationsof toxin, the concentration of toxin associated with each image, and thepercent of cells killed by such concentration may be provided to theevaluation module 218. The evaluation module 218 may compare thepredictions of the model to such experimental data to develop error barsor confidence scores that indicate how well the model predicts theactual experimental data.

Continuing with the above example, experimental data that may beprovided may also include information regarding how many cells in animage of cells exposed to a particular concentration of the toxin wereactually dead and how many cells appeared dead but were actually alive(i.e., false positives). In some embodiments, the evaluation module 118may use such additional data to develop confidence scores that indicatethe ability of the model to predict or take into account false negativesor false positives. In some embodiments, the evaluation module 118 mayconsider situations in which all or most cells of a sample produce acontinuous range of responses. For example, if a drug is known to causecells to produce a particular protein, and at doses a minimal amount ofprotein may be produced and that as the dosage is increased, the amountof protein produced also increases. In such situations, the modeldevelopment module 212 may train the model based on the amount ofresponse (i.e., amount of protein produced) exhibited by cells ratherthan the number of cells that exhibited the response. In someembodiments, cells may have the DNA thereof modified to add afluorescent marker (e.g., Green Fluorescent Protein) to the proteindeveloped by the cells in response to the drug. In these cases, theexpression of the protein may be measured as the protein moves outsidethe nucleus of the cell, and/or groups or combines with other proteins.

In some embodiments, a high value for the confidence score indicatesbetter model performance, i.e., fewer errors between output of the modeland the actual treatment applied to the wells 118A that comprise theevaluation subset. In other embodiments, a low confidence score may beassociated with better model performance, wherein a confidence score ofzero indicates perfect prediction of the treatment applied to the wells118A of the evaluation subset.

In some embodiments, if a high value of the confidence score indicatesbetter model performance and the confidence score exceeds apredetermined threshold value, the model is considered acceptable andthe model may be used to analyze production sample trays 116. Similarlyif a low value of the confidence score indicates better performance andthe confidence score is less than a predetermined threshold value, themodel is a considered acceptable.

In other embodiments, the model evaluation module 218 provides theconfidence score to the user interface module 204 for display on theuser computer 202. In such embodiments, the operator may indicatewhether the model associated with the confidence score is acceptable ornot acceptable.

If the model is not acceptable, the model development module 212 maytune the model by replacing one or more parameters used to develop themodel with other parameters that show a high correlation between theparameter and the wells 118A of the training subset. Machine learningtechniques as described above may be used thereafter to develop a tunedmodel. The tuned model may be evaluated and/or tuned further until amodel is developed that produces an acceptable confidence score whenapplied to the evaluation subset of the wells 118A.

After the model is developed, the model may be stored in the modeldatabase 216 until needed to analyze a production sample tray 116. Themodel may be used with a production sample tray 116 with wells 118 inwhich cells similar to those used to develop model have been depositedand treated with the treatment associated with the model.

In some embodiments, when the production sample tray 116 is disposed inthe HCIS 100, the user interface module 204 may display on the usercomputer 202 a list of models stored in the model database 216.Thereafter, the user interface module 204 may receive from the usercomputer 202 a selected of one of the models stored in the modeldatabase 216. The user interface module 204 provides the selected modelto a model application module 220. The model application module 220determines the parameters required to apply the model, and the imagecapture parameters necessary to obtain values of such parameters. Themodel application module 220 directs the image acquisition module 208 toobtain the necessary images of each well 118 of the production sampletray 116, and uses the parameter calculation module 214 to developvalues of the required parameters. Such developed values of theparameters may then be supplied as inputs to the selected model todetermine if the cells disposed in the well 118 associated with suchparameters are responsive to the treatment associated with the model.

FIG. 5 shows a flowchart 300 of processing that may be undertaken by theresponse analysis system 200 to develop the trained model. At step 302,the user interface module 204 receives reference tray information fromthe user computer 202 and stores such information in the treatmentinformation database 206. At step 304, the image acquisition module 208captures images of each well 118A of the reference tray 116A, and storessuch images in the images database 210.

After the images are stored, the model development module 212 selects atraining subset of the wells 118A, at step 306. At step 308, the modeldevelopment module 212 selects one of the wells 118A of the trainingsubset that has not been analyzed, and provides the images associatedwith such well 118A in the images database 210 to the parametercalculation module 214. At step 310, the parameter calculation module214 analyzes such images, and develops values of various parameters ofsuch images and/or cells identified in such images.

At step 312, the model development module 212 determines if there is anywell 118A of the training subset that has not been analyzed by theparameter calculation module 214. If so, processing proceeds to step308.

Otherwise, at step 314, the model development module 212 correlates theconcentration of treatment supplied to each well 118A of the trainingsubset and the values of parameters development from images associatedwith such well. The model development module 212 uses those parametersthat have the highest correlation to train a model, at step 316.

In one embodiment, all of the parameters that have a correlation greaterthan a predetermined value are used to train the model. If too manyparameters are used, overfitting may occur in which parameters becomecorrelated by happenstance and may not be have a high predictive valuewhen used to evaluate other samples. Increasing the number of samplesused to develop the correlations between the characteristics of theparameters and the treatments applied to the wells 118A may reduce suchoverfitting.

At step 318, the model evaluation module 218 selects a subset of thewells 118A of the reference sample tray 116A as an evaluation subset. Atstep 320, the model evaluation module 218 determines which parameters ofeach image are required to apply the model developed at step 316. In oneembodiment, each parameter may be evaluated to determine how well suchparameter by itself (i.e., without other parameters) is correlated withthe amount of treatment applied to each well 118A of the referencesample tray 116A. Individual parameters that exhibit a high correlationwithout other parameters may be used to train the model. After the modelis trained, additional parameters may be added or modified to improvethe accuracy of the model. In other embodiments, a predetermined set ofparameters may be used initially to train the model, and then additionalparameters may be added to tune the model.

At step 322, the model evaluation module 218 selects one of the wells118A of the evaluation subset that has not been analyzed, and providesthe images from the images database 210 associated with the selectedwell and the parameters identified at step 320 to the parametercalculation module 214. At step 324, the parameter calculation module214 analyzes such images and calculates the values of the identifiedparameters, and supplies the calculated values to the model evaluationmodule 218.

At step 326, the model evaluation module 218 uses the values of theparameters calculated from the images of the selected well 118A asinputs to the model to develop a prediction of whether any cells in theselected well 118A of the evaluation subset has a phenotypical responseto the treatment, and, if so, the concentration of such treatment.Thereafter, the model evaluation module 218, at step 328, compares thepredicted response from the model with the treatment informationassociated with the selected well 118A in the treatment informationdatabase 206, and develops an effectiveness score for the selected well118A. The effectiveness score for the selected well 118A may bedeveloped in a manner similar to that described above to develop theconfidence score.

At step 330, the model evaluation module 218 determines if any wells118A of the evaluation subset remain to be analyzed, and if so, proceedsto step 322. Otherwise, at step 332, the model evaluation module 218aggregates the effectiveness scores for all of the wells 118A of theevaluation subset to develop an effectiveness score for the model.

At step 334, the model evaluation module 218 determines if theeffectiveness score for the model is sufficient, for example, bycomparing such score with a predetermined threshold or asking theoperator. If the model effectiveness score is sufficient, then the modelevaluation module 218 stores the model in the model database 216. Whenstoring the model the operator may be asked, via the user interfacemodule 204 to enter information that may be used to identify the model.Such information may include, for example, a name for the model, celltype and treatment type associated with the model, the operator whocreated the model, and the like. Such operator supplied information andautomatically generated information, such as date when the model wascreated and the parameters that are necessary to run the model, arestored with the model in the model database 216.

If the model effectiveness score is not sufficient, then, at step 334,the model development module 212 modifies the parameters associated withinputs of the model and proceeds to step 316 to train the adjustedmodel. In some embodiments, there may be limits as to the numbertraining iterations that may be used to train the adjusted model beforea determination is made that the model is not converging. Suchdetermination may be presented to the user and the user may request thatfurther iterations be used or that training be stopped.

The model stored in the model database 216 may be retrieved thereafterto automatically process a production sample tray 116.

FIG. 6 shows a flowchart 350 of the processing undertaken by theresponse analysis system 200 to process the production sample tray 116.At step 352, the model application module 220 communicates with thecontroller 112 of the HCIS 100 to confirm that the production sampletray 116 is loaded on the X-Y stage 102.

At step 354, the model application module 220 retrieves a list of modelsstored in the model database 216. The list of models and identifyinginformation associated with each model are provided to the userinterface module 204 for display on the user computer 202. Thereafter,the operator is asked to select the model. The user interface module 204receives such selection and provides the selection to the modelapplication module 220, also at step 354.

At step 356, the image acquisition module 208 directs the HCIS 100 tocapture images of the wells 118 of the production sample tray 116, andstores such images in the images database 210.

At step 358, the model application module 220 queries the model database216 to identify the parameters that are necessary to run the model. Atstep 360, the model application module 220 selects a well 118 of theproduction tray and provides the identified parameters and the imagesassociated with such well 118 to the parameter calculation module 214.

At step 362, the parameter calculation module 214 calculates values ofeach of the parameters identified at step 358. At step 364 the modelevaluation module 218 uses such calculated values as inputs to themodel, and determines an output that indicates whether the cellsdisposed in the well 118 selected at step 360 exhibit a phenotypicalresponse to the treatment, and, if so, the concentration of thetreatment associated with such response.

Such output is added to a list of results, at step 368, that includes anentry for each well 118 of the production sample tray 116, and eachentry includes the output of applying the model to the images associatedwith such well 118.

At step 370, the model application module 220 determines if all of thewells 118 of the production sample tray 116 have been processed usingthe model. If any wells 118 remain, the model application module 220proceeds to step 360. Otherwise, at step 372, the model applicationmodule 220 provides the results to user interface module 204 for displayon the user computer 202, stores such results in a data store (notshown) that may be accessed by the operator, and/or transmits suchresults to another system (not shown) for further analysis.

The response analysis system 200 may be used to determine how cellsrespond to different doses of treatment, to evaluate the effect apharmaceutical product has on cells, or the types of cells that may beaffected by a pharmaceutical product. The response analysis system 200may help a researcher identify the types of cells that respond to thetreatment or to identify commonality in cells that are responsive to aparticular treatment. In addition, the response analysis system 200 mayallow the operator to identify which treatment or dose of a treatment iseffective in killing certain types of cells (e.g., cancer cells) but notother types of cells. Further, the response analysis system 200 may beused to identify particularly hearty cells that may be appropriate forcloning. The response analysis system 200 may also be evaluate thephenotypical response of organelles within cells in an image.

As noted above, the response analysis system 200 may be used to evaluateor identify the phenotypical response of a particular type of cell in ahetrogenous population of cells disposed in the well 118 of a sampletray. For example, the phenotypcal response to a treatment of neuronsmay be evaluated in a population that includes neurons and glial cells,or even the phenotypical response of live cells in a population thatincludes live cells and dead cells.

Different models developed as described above may be used in combinationto identify different phenotypical responses to treatments applied towells 118 of a sample tray 116. For example, a first model associatedwith a treatment of cells may be applied to images of wells 118 of aparticular sample tray 116 to evaluate the phenotypical response of thecells in such wells 118 to the treatment. A second model associated witha different treatment then may be applied to the images of the wells 118of the same sample tray 116 to evaluate the phenotypical response of thecells in the wells 118 to the second treatment. The first model and thesecond model may be used to evaluate the phenotypical response ofidentical types of cells or different types of cells. In someembodiments, the first model may be used to evaluate the phenotypicalresponse of live cells and the second model may be used to evaluate thenumber of deceased cells in the wells 118 of the sample tray 116.

As noted above, different models developed may be used in combination(i.e., multiplexed) to identify multiple responses of individual cellsor to identify responses of individual cells. In some embodiments, thecombined models may be used to identify the percent of healthy cellsthat are responsive to the treatment, the number of cells undergoingcell division that are producing a certain protein, the number ofinfected cells (e.g., infected with bacteria or virus) that areabsorbing a particular molecule, and the like.

In some embodiments, the response analysis system 200 may be used todevelop a model that is trained with images of wells 118 treated with adrug that has a known effect on cells but also has an undesirablecharacteristic as a pharmaceutical agent. The model may be used toevaluate the effects on cells of other drug candidates. In anotherapplication, the response analysis system 200 may be used to determinehow cells respond to known agents after being modified throughtreatment.

Although the response analysis system 200 above is described in thecontext of identifying cells in an image, it should be apparent thatsuch system may be used identify the response of any type of object ofwhich an image may be acquired.

Although the response analysis system 200 has been described above withrespect to samples of cells deposited in wells 118 of a tray 116, itshould be apparent that the response analysis system 200 may be used toautomatically analyze the phenotypical response of cells provided inother types of carriers. For example, instead of a well 118 of a tray116 in which a particular amount of treatment has been applied to eachwell 118, the cells may be provided on a set of slides in which aparticular amount of treatment has been applied to each slide.Similarly, the cells may be provided in discrete samples of fluids in aflow cytometry system, and each such sample of fluid may be analyzed asthe well 118 is analyzed in the description above.

It will be understood and appreciated that one or more of the modules,processes, sub-processes, and process steps described in connection withFIGS. 1-6 may be performed by hardware, software, or a combination ofhardware and software on one or more electronic or digitally-controlleddevices. The software may reside in a software memory (not shown) in asuitable electronic processing component or system such as, for example,one or more of the functional systems, controllers, devices, components,modules, or sub-modules schematically depicted in FIGS. 1-6. Thesoftware memory may include an ordered listing of executableinstructions for implementing logical functions (that is, “logic” thatmay be implemented in digital form such as digital circuitry or sourcecode, or in analog form such as analog source such as an analogelectrical, sound, or video signal). The instructions may be executedwithin a processing module or controller (e.g., the user interfacemodule 204, the image acquisition module 208, the model developmentmodule 212, the parameter calculation module 214, the model evaluationmodule 218, and the model application module 220), which includes, forexample, one or more microprocessors, general purpose processors,combinations of processors, digital signal processors (DSPs), fieldprogrammable gate arrays (FPGAs), or application-specific integratedcircuits (ASICs). Further, the schematic diagrams describe a logicaldivision of functions having physical (hardware and/or software)implementations that are not limited by architecture or the physicallayout of the functions. The example systems described in thisapplication may be implemented in a variety of configurations andoperate as hardware/software components in a single hardware/softwareunit, or in separate hardware/software units.

The executable instructions may be implemented as a computer programproduct having instructions stored therein which, when executed by aprocessing module of an electronic system, direct the electronic systemto carry out the instructions. The computer program product may beselectively embodied in any non-transitory computer-readable storagemedium for use by or in connection with an instruction execution system,apparatus, or device, such as a electronic computer-based system,processor-containing system, or other system that may selectively fetchthe instructions from the instruction execution system, apparatus, ordevice and execute the instructions. In the context of this document,computer-readable storage medium is any non-transitory means that maystore the program for use by or in connection with the instructionexecution system, apparatus, or device. The non-transitorycomputer-readable storage medium may selectively be, for example, anelectronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device. A non-exhaustive list ofmore specific examples of non-transitory computer readable mediainclude: an electrical connection having one or more wires (electronic);a portable computer diskette (magnetic); a random access, i.e.,volatile, memory (electronic); a read-only memory (electronic); anerasable programmable read only memory such as, for example, Flashmemory (electronic); a compact disc memory such as, for example, CD-ROM,CD-R, CD-RW (optical); and digital versatile disc memory, i.e., DVD(optical). Note that the non-transitory computer-readable storage mediummay even be paper or another suitable medium upon which the program isprinted, as the program can be electronically captured via, forinstance, optical scanning of the paper or other medium, then compiled,interpreted, or otherwise processed in a suitable manner if necessary,and then stored in a computer memory or machine memory.

It will also be understood that receiving and transmitting of data asused in this document means that two or more systems, devices,components, modules, or sub-modules are capable of communicating witheach other via signals that travel over some type of signal path. Thesignals may be communication, power, data, or energy signals, which maycommunicate information, power, or energy from a first system, device,component, module, or sub-module to a second system, device, component,module, or sub-module along a signal path between the first and secondsystem, device, component, module, or sub-module. The signal paths mayinclude physical, electrical, magnetic, electromagnetic,electrochemical, optical, wired, or wireless connections. The signalpaths may also include additional systems, devices, components, modules,or sub-modules between the first and second system, device, component,module, or sub-module.

INDUSTRIAL APPLICABILITY

The use of the terms “a” and “an” and “the” and similar references inthe context of describing the invention (especially in the context ofthe following claims) are to be construed to cover both the singular andthe plural, unless otherwise indicated herein or clearly contradicted bycontext. All methods described herein can be performed in any suitableorder unless otherwise indicated herein or otherwise clearlycontradicted by context. The use of any and all examples, or exemplarylanguage (e.g., “such as”) provided herein, is intended merely to betterilluminate the disclosure and does not pose a limitation on the scope ofthe disclosure unless otherwise claimed. No language in thespecification should be construed as indicating any non-claimed elementas essential to the practice of the disclosure.

Numerous modifications to the present disclosure will be apparent tothose skilled in the art in view of the foregoing description. It shouldbe understood that the illustrated embodiments are exemplary only, andshould not be taken as limiting the scope of the disclosure.

We claim:
 1. A system to analyze a phenotypical response of a cell to atreatment, comprising: an image capture device; and a response analysissystem that includes computer-executable code stored on one or morenon-transitory storage devices that, when executed, causes one or moreprocessors to: receive images of a plurality of reference cell carriersand treatment information associated with the plurality of referencecell carriers, identify parameters of images of the plurality ofreference cell carriers based on the treatment information, theidentified parameters distinguishing those cells in the plurality ofreference cell carriers to which the treatment has been applied fromother cells in the plurality of reference cell carriers, train a modelusing the identified parameters, receive from the image capture devicean image of a cell carrier to be evaluated, and apply the trained modelto the image of the cell carrier to be evaluated to indicate a responselevel of a cell in the cell carrier to be evaluated.
 2. The system ofclaim 1, wherein at least one of the identified parameters is aparameter associated with a representation of a cell in one of theimages.
 3. The system of claim 2, wherein the parameter associated withthe representation of the cell includes one of: distance between an edgeand a nucleus of the representation of the cell, a shape of therepresentation of the cell, and variation in intensity of pixelsassociated with the representation of the cell.
 4. The system of claim2, wherein the response analysis system develops a plurality ofparameters of the images of the plurality of reference cell carriers,and the identified parameters are a subset of the plurality ofparameters.
 5. The system of claim 4, wherein at least one of theplurality of parameters is developed by analyzing a plurality of imagesof a sample area of a particular cell carrier.
 6. The system of claim 5,wherein the plurality images of the sample area are taken usingdifferent illumination conditions.
 7. The system of claim 4, wherein theresponse analysis system calculates a correlation between each of theplurality of parameters and a concentration of treatment applied to thereference cell carriers and the identified parameters are selected inaccordance with such correlation.
 8. The system of claim 1, wherein themodel is an artificial neural network and each identified parameter isassociated with an input node of the artificial neural network.
 9. Thesystem of claim 1, wherein the response analysis system uses one of:deep learning, forest try, random forest, genetic algorithm,metaheuristic, k-means clustering, ant swarming, ant colonyoptimization, and simulated annealing techniques to train the model. 10.The system of claim 1, wherein the response analysis system developsfrom the image of the cell carrier to be evaluated values of theidentified parameters and provides the developed values as an input tothe model.
 11. The system of claim 1, wherein one of the identifiedparameters is one of: an average pixel intensity, pixel coordinateswhere cell nuclei are located, strength of pixel intensity variation atedges of cells, an orientation of edges relative to nuclei of cells inan image, a measure of texture or variation of pixel intensities atdifferent magnifications, a standard deviation of pixel intensity,frequencies associated with peaks of a Fourier transform, a waveletmatch, and Laplacian of Gaussian peak.
 12. The system of claim 1,wherein the response analysis system identifies in each image of theplurality of reference cell carriers one or more pixels associated witha cell represented in the image.
 13. A method to analyze a phenotypicalresponse of a cell to a treatment, comprising the steps of: receivingimages of a plurality of reference cell carriers and treatmentinformation associated with the plurality of reference cell carriers;identifying parameters of the images and/or cells represented in theimages based on the treatment information, the identified parametersdistinguishing those cells in the plurality of reference cell carriersto which the treatment has been applied from other cells in theplurality of reference cell carriers; training a model using theidentified parameters; receiving from an image capture device an imageof a cell carrier to be evaluated; and applying the trained model to theimage of the cell carrier to be evaluated to indicate a response levelof a cell in the cell carrier to be evaluated.
 14. The method of claim13, wherein the step of identifying parameters includes the steps ofdeveloping a plurality of parameters of the images and/or cellsrepresented in the images, calculating a correlation between each of theplurality of parameters and a concentration of treatment applied to thereference cell carriers, and identifying a subset of the plurality ofparameters in accordance with such correlation.
 15. The method of claim14, wherein at least two of the plurality of images are of a sample areataken under at least two different illumination conditions.
 16. Themethod of claim 15, wherein the step of training the model comprises thestep of training an artificial neural network.
 17. The method of claim16, wherein the step of training the artificial neural network includesthe step of associating an identified parameter with an input node ofthe artificial neural network.
 18. The method of claim 13, wherein thestep of training the model includes the step of training the model withone of: deep learning, forest try, random forest, genetic algorithm,metaheuristic, k-means clustering, ant swarming, ant colonyoptimization, and simulated annealing techniques.
 19. The method ofclaim 13, wherein the step of identifying the parameters includes thestep of identifying pixels associated a representation of a cell in oneof the images of the plurality of reference carriers.